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Using an optimization variational Monte Carlo method, we study the half-filled-band Hubbard model on 
anisotropic triangular lattices, as a continuation of the preceding study [J. Phys. Soc. Jpn. 75 (2006) 074707]. 
We introduce two new trial states: (i) A coexisting state of (n, 7r)-antiferromagnetic (AF) and a d-wave singlet 
gaps, in which we allow for a band renormalization effect, and (ii) a state with an AF order of 120° spin structure. 
In both states, a first-order metal-to-insulator transition occurs at smaller U /t than that of the pure d-wave state. 
In insulating regimes, magnetic orders always exist; an ordinary (7r,7r)-AF order survives up to t' /t ~ 0.9 
(U/t = 12), and a 120°-AF order becomes dominant for t' /t ^ 0.9. The regimes of the robust superconductor 
and of the nonmagnetic insulator the preceding study proposed give way to these magnetic domains. 

PACS numbers: 74.70.-b, 74.20.-z 



I. INTRODUCTION 

A series of k-(BEDT-TTF)2X [k-ET salts] have intrigu- 
ing properties specific to strongly-coiTelated systems; they of- 
ten undergo unconventional superconductor (SC)-to-insulator 
transitions through the chemical substitution of X or un- 
der applied pressure, and have good two-dimensionality in 
conductivity with frustrated lattice structure. As a model 
of these compounds, the half-filled-band Hubbard model on 
anisotropic triangular lattices 01] (an extended square lattice 
with hopping integral t in x and y directions, and t' in one 
diagonal direction [1,1]) has been intensively studied; ||2J 

W = X! ^kCkffCkCT + U^Ui^n^i, (1) 

kcr i 

where ek = —2t{coskx + cosky) — 2t' cos{kx + ky), and 
U, t, t' > 0. To clarify the properties of this model in the 
strongly-correlated region, U/t ^ 1, especially Mott transi- 
tions, reliable theoretical approaches are needed. To this end, 
the present authors recently applied to eq. ([T]) an optimiza- 
tion (or coiTelated) variational Monte Carlo (VMC) method, 
which can deal with SC and a Mott transition as a continu- 
ous function of U/t. Henceforce, we call this preceding study 
'©'•iHl In (I), we chiefly considered various properties of the 
d^2_y2-'wave singlet state, 'i'g, and constructed a ground-state 
phase diagram in the t'-U plane, by comparing its energy with 
that of the ordinary (tt, 7r)-antifeiTomagnetic (AF) state, 
Most of the results are consistent with the behavior of k-ET 
salts, but the area of an (tt. 7r)-AF insulator is unexpectedly 
limited (t' /t ^ 0.4), in considering the appearance of the AF 
order in e.g. k-(ET)2CuN(CN)2C1 (t'/t - 0.74), as well 
as the vanishing point of the AF order expected in the J- J' 
Heisenberg model (t'/t ^ 0.8) ifioll . As we pointed out in (I), 
this disagreement possibly stems from the fact that the rf-wave 
singlet state and the AF state were treated separately; thereby, 
the former state does not include a seed of an AF long-range 
order, and the latter a band renormalization effect. 



For t' t, many theoretical studies flU, H, 0, H] for the 
Hubbard model have obtained results of dominant nonmag- 
netic insulating state, which are consistent with the insulat- 
ing state found in k-(ET)2Cu2(CN)3 with t'/t - 1.06 lH. 
Nonetheless, we should be also concerned about the AF order 
with 120-degree spin structure, which is considered to prevail 
in the isotropic case of the J-J' Heisenberg model. jioi [Till 
Actually, a recent VMC studyfl^ for a <-J-type model on 
the isotropic triangular lattice concluded that the 120°-AF or- 
dered state is dominant in an unexpectedly wide range of dop- 
ing rate. Thus, it is possible that the 120°-AF order is robust 
also in the Hubbard model with t' ^ t and sufficiently small 
values of U /t for the organics. 

In this paper, as a continuation of (I), we introduce two trial 
states; (i) A state which includes (tt, 7r)-AF and d-wave gaps 
simultaneouslv; lll3l [3 and then a band (or Fermi-surface) 
renormalization effect owing to the electron correlation is 
taken into account, ifisll (ii) A state which exhibits the 120°- 
AF order. In addition to these functions, we newly con- 
sider SC states with pairing symmetries suitable for t' > t. 
Our main interest here is the competition among these states 
and those treated in (I). It is found that first-order metal-to- 
insulator transitions always occur at smaller values of U/t 
than those for the pure d-wave state. In the insulating regime, 
the (tt, 7r)-AF order remains up to t'/t ~ 0.9, owing to the 
band renormalization effect we considered in the coexisting 
state, and the 120°-AF order becomes predominant in the 
range of t'/t ^ 0.9. Consequently, a magnetic order, namely 
the (tt, 7r)-AF or 120°-AF order, always exists in the insulat- 
ing regime, and a regime of a nonmagnetic insulator vanishes. 
In addition, a domain of dominant SC found in (I) disappears 
within the present results. The previous phase diagram is sub- 
stantially modified. 

In ini we explain the trial wave functions used, and reca- 
pitulate the main points of (I) as a motivation of this study. 
InUm we represent the VMC results. In|IVl we briefly sum- 
marize this study, and compare with experimental and other 
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theoretical results. 

A part of the results have been reported before. lfl6l 



II. WAVE FUNCTIONS 

As usual, we use Jastrow-type trial wave functions: ^ = 
V^, in which $ denotes a one-body (Hartree-Fock) part ex- 
pressed as a Slater determinant, and V a many-body corre- 
lation factor In III Al we describe the correlation factor V. 
In IIIBI we point out insufficient points in the wave functions 
used in (I), and introduce a coexisting state of the (tt, 7r)-AF 
and d-wave gaps in which the one-body band structure is mod- 
ified by optimizing a hopping parameter t', as renormalization 
owing to electron correlation. In III CI we formulate a state 
with an AF order of 120° spin structure, as an another new 
trial state. In lllDI we briefly touch on the conditions of the 
VMC calculations. 



A. Correlation factor 

When one treats the Hubbard model on the basis of a var- 
iational method, it is crucial to introduce, in addition to the 
well-known Gutzwiller (onsite) factor Vg, iflTl [isll intersite 
correlation factors lfl9ll20ll into Jastrow-type wave functions. 
In particular, near half filling, the binding effect of a doubly- 
occupied site (doublon) to an empty site (holon) is indispens- 
able to describe a Mott transition as well as various quantities 
ap prop riately. Iil9il To this end, we have repeatedly studied 
ll3i l2lll22ll23ll a four-body factor formally written as. 
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in which di = miriii, = (1 — ni|)(l — n^;), and t (t') 
runs over all the adjacent sites in the bond directions of t (t'). 
In eq. (|2]i, fi {fj!) is a variational parameter which controls the 
binding strength between a doublon and a holon in the bond 
direction t it'). We have confirmed that Vq works effectively 
in the model, eq. ©-ill 



B. Coexisting state of d-wave and AF gaps 

Using V = VqVg, we mainly studied, in (I), a dj;2_y2- 
wave singlet state: ^'q — V^d, where is the BCS function 
with a (ij.2_j,2-wave gap: 



Hamiltonian eq. ([T]l- In (I), we obtained the following results 
within 'I'g. (i) A first-order Mott (conductor-to-nonmagnetic- 
insulator) transition takes place for arbitrary t' /t at U = Uc 
roughly of the bandwidth. This transition is induced by the 
binding (and unbinding) of a doublon (negatively charged) to 
a holon (positively charged), unlike the famous Brinkman- 
Rice transition. [24] (ii) Robust d-wave SC appears in a re- 
stricted parameter range immediately below Uc and of weak 
frustration (t' /t ^ 0.7). This SC is considered to be induced 
by a short-range (tt, 7r)-AF spin correlation, because when- 
ever the superconducting (SC) correlation function is sizably 
enhanced, the spin structure factor S'(q) has a sharp peak at 
the AF wave number, q = K = (tt, tt). (iii) In the insulating 
regime, '^q exhibits a spin-gap behavior and does not have an 
(tt, 7r)-AF long-range order, although S{q) has a sharp peak 
at q = K, namely a short-range AF correlation considerably 
develops. 

To consider the competition between and a state with 
the (tt, 7r)-AF long-range order [see Fig.fTfa)], which should 
prevail for small t' /t, we also studied a projected AF state, 
= V^AF, where (E>af is a mean-field-type (tt, 7r)-AF 
state. In ^'g^, we did not renormalize F, because the varia- 
tional energy E to be minimized becomes a discrete function 
of i' /t. We found that (iv) the stable range of ^fg^ against 
^fg is restricted to a weakly frustrated regime, t' /t ^ 0.4 (for 
U/t = 6), and this range tends to shrink as U/t increases. As 
notified in (I), the above results (iii) and (iv) are not consis- 
tent with various approximate results ilOfI for the correspond- 
ing J- J' spin model, which predict that the (tt, 7r)-AF domain 
continues up to t' /t ^ 0.8. To resolve this disagreement, a 
seed of the AF order should be introduced into ^'g, and the 
renormalization of owing to U into "fg^. 

In this paper, we study a wave function, ^fg = V^co, 
which meets the above requirements by merging ^fg and 
y^^. In ^'q, the d-wave gap and an AF order can coexist. 
|[l3|] The one-body part is written as. 
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in which N,, is the electron number, and (/Sk is the ratio of BCS 
coefficients: 

Vk Ak 

= — 

Uk 

with 



ek-C+x/(£k-C)2 + A^' 



(6) 



Ek = — 2t(cos kx + cos ky) — 2t' cos{kx + ky), (7) 

and 6^ is a creation operator that diagonalizes the ordinary 
(tt, 7r)-AF Hartree-Fock Hamiltonian, and is given as 

^f^A+K.a: (8) 



"kcL 



Ak = Arf(cos kx — cos ky). 



(4) 



6^ 

"k+K,o- 



-^/3k4^^ + akci^+K.o 



(9) 



In \I/g, we allow for renormalization of the one-body band Ek 
owing to electron correlation, by varying t'{= t') in ^d as 
a variational parameter, [15] independently of t' fixed in the 
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FIG. 1; (Color online) Schematic representation of spin structure in 
two AF orders studied in this paper for the anisotropic triangular lat- 
tice: (a) an ordinary (tt, 7r)-AF order, and (b) an AF order with 120° 
spin structure. 



with 7k — —2t(coskr^ + cosky) and = +(— )1, according 
as (T = t (I). In addition to the six parameters in VPg, namely, 
g [Gutzwiller (onsite) parameter], /i, fi', A^, ( (chemical po- 
tential) and i', has the seventh parameter Aaf^ which 
controls the staggered spin field and is closely connected to 
the (tt, 7r)-AF order parameter TOs (sublattice magnetization). 
Note that, in contrast to Aaf, a finite optimized value of A^ 
does not necessarily mean that a SC gap opens, but an insu- 
lating spin gap. For Aaf (Ad) 0, is reduced to 
(^Q^)- Thus, we may regard ^'g as VPg in which the (tt, tt)- 

,AF 



AF long-range order can arise, and also as VPg into which a 
band renormalization effect is introduced through the d-wave 
gap- 



C. AF-ordered state with 120-degree spin structure 

As discussed in|I] an AF-ordered state with 120° spin struc- 
ture [see Fig.[nb)] is plausible for the region of t' /t ^ 1. We 
introduce such a state, vE'i2o = ^-'$120, for the Hubbard model 
eq. ([Til, and check its stability for finite values of U/t and con- 
sistency with the results obtained for U/t — 00. poilTTl] 

As the one-body part, $120, we use a Hartree-Fock ground 
state for the Hamiltonian eq. ([T]i. As explained in Fig. |2] we 
consider six sublattices (A-F); the spin quantization axis of 
a sublattice is turned by 60 degrees from that of a neighbor- 
ing sublattice. Using this scheme, the Hamiltonian eq. ([T) is 
transformed to 



H 



<«AJA + 1> \ t / 



+ t' 



E 

(«A jA + 2) 



(«Lt"1,j 



3/ \«iA + 2,i 



h.c. 



FIG. 2: (Color online) Schematic explanation of Hartree-Fock ap- 
proximation for an AF order with 120° spin structure. In (a), it is 
shown how we divide the anisotropic triangular lattice into six sub- 
lattices (A-F) with different directions of a spin quantization axis, 
which are illustrated in (b): the axis of B (C,D,E,F,A) sublattice is 
obtained by turning that of A (B,C,D,E,F) sublattice by 60 degrees. 
For these sublattices, we suppose that the gap parameter is staggered, 
namely A120, — A120, A120, • • • , leading to the formation of a 120°- 
AF order in Fig.[TJb). 



where 



cos{9) -sm{9) 
sm{e) coa{e) 



(12) 



I, a 



is a creation operator in the sublattice representation, 

= a] ^tti^a, A (=A-F) is a sublattice index, ix runs over all 
the sites on sublattice A, and an angle (round) bracket in the 
summation indices in eq. ( fTTT i indicates a nearest(diagonal)- 
neighbor pair. We apply a Hartree-Fock decoupling to the in- 
teraction term in eq. ( fTTT ). 



and assume that the gap is staggered as 



const., 
(13) 
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to form a 120°-AF order. Using the operators for sublattices, 
the Hartree-Fock Hamiltonian in the wave-number represen- 
tation is given as. 
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where ai^^ is the Fourier transformation of a\ 

K,(T 'iA,( 



Ai = -tcos(7r/6)(e-*'=== +6"*'^'' 

A2 = -<'cos(7r/3)e-*('=-+'=«), 

Bi+ = tsin(7r/6)(e"''''- +e~*'=«), 

Bi_ = -tsin(7r/6)(e'''- +e*'=«), 

B2+ = t'sm{TT/3)e-''^''-+''y\ 

B2- = -t'sm{TT/3)e''-''-+''y\ 



(15) 



and 



(16) 



As $120, we adopt the lowest-energy eigenvector obtained by 
diagonalizing eq. ( fTSl) . However, we do not determine A120 
by a self-consistent equation in the Hartree-Fock approxima- 
tion, but optimize A120 as a variational parameter in ^'q 
simultaneously with the other parameters with respect to the 
original Hamiltonian eq. ( fTTT ). If A120 is finite, all sublattices 
have staggered spin densities, constituting the 120° spin struc- 
ture. 



D. Variational Monte Carlo calculations 
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Generally, it is not easy to accurately calculate expecta- 
tion values of a many -body wave function with analytic ap- 
proaches. Here, we apply an optimization VMC method, llzsll 
which effectively minimizes the variational energy and makes 
a virtually accurate evaluation, to the wave functions men- 
tioned in this section. We have performed VMC calculations 
mainly for the lattice of TVg = L x L sites with L — IQ and 
12. The conditions of calculations here are mostly the same 
as those in (I). 

m. RESULTS 

In nil Al we consider the energies of an d ^'g °, and the 
critical behaviors appearing in them. In llllBI we show these 
critical behaviors indicate a metal-to-insulator transition. In 
unci we discuss the properties of the AF order in the insulat- 
ing regime of ^fg, and the eventual phase diagram. In llllDI 
we consider the BCS state with another pairing symmetries 
expected for the region of t' ^ t. 

A. Stability of coexisting state and 120° -AF state 

We start with the energy reduction of the coexisting state 
^-g" and the 120°-AF state fi^o for t' - t. In Figs. Ha) 



FIG. 3: (Color online) Total energies of the coexisting state ^'q 
[d+AF], the 120° -AF state [120AF], and the d-wave state 
[d] are compared as a function of the correlation strength, for (a) 
t'/t — 0.8 and (b) 1.0. The critical values of Mott transitions Uc/t 
are indicated by arrows for respective states. Although the data for 
L = 10 and 12 are plotted, the system-size dependence is almost 
negligible in this scale. For t'/t = 1, the system of L = 10 is not 
used because the closed-shell condition is not satisfied. 



and[3lb), the total energy per site E is compared among '^q 
{E""), ^-i^o (i;i20) and 5-^ {E'^) for t'/t = 0.8 and 1.0, re- 
spectively. For both values of t'/t, the curves of E/t for the 
three states are almost indistinguishable from one another for 
small U /t, whereas they separate with cusps as U/ 1 becomes 
large. In fact, as we will see shortly, these cusps indicate 
metal-insulator transitions. For t'/t = 0.8, E"^" exhibits a 
cusp first at J7 = — 6.65t ± O.OSi and becomes appre- 
ciably lower than both E'^ and E^^" for U > U^°. On the 
other hand, for t'/t = 1.0, E-^-^^ exhibits a cusp first at C/ = 
[/120 = 7.65i ± o.05i and becomes the lowest for U > U^'^°. 
Thus, the lowest-energy state for large U /t is switched from 
^-g" to ^^20 jjjg j^jjgg of 0.8 < t'/t < 1.0. To see t'/t 
dependence of E/t in the insulating regime (U > Uc), we 
plot the total energies at U/t = 12 of various states in Fig.|4] 
For t' < t'^ ^ 0.90i, the coexisting state is the most stable. 
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FIG. 4: (Color online) Comparison of total energies in the insulat- 
ing regime (U/t — 12) as a function of t' /t among various states: 
a coexisting state 'Pg' [d+AF], a 120° -AF state "P^^" [120AF] and 



three singlet states: a d wave 



an ext.s+dxy wave 



s+d' 



[ext.s+dxy] and a d^2_y2+idxy wave '5^+"' [d+jd]. The latter two 
states will be discussed in 1111 Dl The arrows indicate the bound- 
ary values between t'^/t and tc2/i satisfying E^^^ = E'^° and 



^120 ^ 



, respectively. 



and the decrease in E/t from estimated in (I) is approx- 
imately 7.6%, iiTespective of the value of t' /t. This invariant 
behavior of E/t with respect to /t is caused by marked band 
renormalization; this point will be discussed in detail in llllCI 
In contrast, decreases rapidly as t'/t increases, and be- 
comes the lowest for t' > t'^i. As expected, ^'g*' becomes 
predominant near the symmetric point (t' /t ~ 1). Conse- 
quently, the area where the pure d-wave singlet state pre- 
vails does not appear in the insulating regime. 

To discuss the energy reduction more closely, especially in 
the conductive regime, we introduce the condensation energy: 




Er^E' 



E, 



(17) 



FIG. 5: (Color online) Comparison of the condensation energy Ec/t 
among (d-nAF), ^Jj^" (120AF) and (d), for (a) t' /t = 0.6, 
(b) 0.8 and (c) 1.0. The arrow on the horizontal axis in each panel 
indicates the critical point of the metal-insulator transition arising at 
the smallest Urmc (= Urmc) among those states. 



where E^ denotes the energy per site of the projected Fermi 
sea, vI/F = P$F, as the reference value. In Fig.|5] E^°, E^'^° 
and E^ are shown for three values of t' /t. Note that E^ for 
every state is almost zero for U < [/"'", where U"^"^/t is 
shown by an arrow in each panel. This means that every state 
for U < U™™ is almost reduced to a normal metallic state 
^g. Here, it is important to recall that, as discussed in (I'). lE6ll 
robust SC occurs only for U^^^^^^ < U < U^, in which Ec/t 
has a small but perceptible finite value. Although this ten- 
dency can be seen in E^/t for: t' /t ~ 0.6 and 6;$C//t<7.15 
[Fig- Ha)], more stable 5'g covers the whole range of SC, 
namely, C/™'" — 17^° < t^onsot- Consequently, comes to 
have no chance to arise appreciable SC. We wiU return to this 
subject in nil B I 



B. Metal-insulator transitions 

In this subsection, we study the critical behavior at [/ = Uc 
found in £"=° and E^^^ (cusps) in Fig.[3]and in E^° and E^^° 
(sudden increases) in Fig. |5] Although we have not men- 
tioned, in fact, E'^° and £'^^" in Fig.[3]undergo clear hysteresis 
(dual-minimum behavior) near the cusps at Uc- This indicates 
a kind of first-order transition takes place at Uc- We will reveal 
the properties of this transition with various quantities. 

First, we take up the doublon density, 

D=±y{n,.n,A = —^^^, (18) 

i 

where Hint denotes the second (interaction) term of the 
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FIG. 6: (Color online) Density of doubly-occupied site (doublon) as 
a function of U jt for the lowest-energy states: the coexisting state 
VpQ for t' It = 0.4-0.8, and the 120° -AF state tjj^" for t'/t = 1.0. 
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FIG. 7: (Color online) The momentum distribution function of the 
lowese-energy states is shown for various values of U jt along the 
path (0, 0)-(7r, 0)-(7r, -7r)-(0, 0) in the Brillouin zone, (a) for il jt = 
0.8 (coexisting state W^, Uc/t ~ 6.65) and (b) for t' /t = 1.0 
(120°-AF state -^^^^ , Uc/t ~ 7.65). The open (solid) symbols de- 
note the data for U < Uc (U > (7c). 



Hamiltonian eq. ([T]). D is regarded as the order parameter 
of metal-insulator transitions, I27l1 by analogy with the parti- 
cle density in gas-liquid transitions. As shown in Fig. |6l D 
exhibits a discontinuity atU — Uc for each t' /t, strongly sug- 
gesting a first-order metal-insulator transition. 
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FIG. 8: (Color online) The charge structure factor A'^(q) for the same 
states with those in Fig. |7] is plotted along the same path: (a) Co- 
existing state for t' /t = 0.8, and (b) 120° -AF state for 
t' /t — 1.0. The open (solid) symbols denote the data for U < Uc 
(U > Uc). 



In Fig. |2l the momentum distribution function, 

'^{k) = ^E<^Lck.), (19) 

of the lowest-energy states is plotted for t' /t = 0.8 (5'q) and 
1.0 (^Q^°). Discontinuities of n(k) at kp in both sections, 
(0,0)-(0,7r) and (0, 0)-(7r, tt), are obvious for U < for 
both magnetic states, whereas n{k) becomes smooth in both 
sections for U > Uc- Because the quasi-Fermi surface van- 
ishes for U > Uc, we may consider that the state becomes 
non-metallic. 

In Fig. [8] we depict the charge structure factor. 



(20) 



with Ni = 7T.il + nil, same states as those in Fig. |7] 

Similarly to the case of 'i'g studied in (I), A^(q) near the F 
point (0, 0) seems linear in |q| for U < Uc, whereas the be- 
haviors of A^(q) abruptly change to roughly quadratic in |q| 
for U > Uc, regardless of vp™ or ^1^20 follows that the 
states are gapless in the charge sector and are conductive for 
U < Uc, but a charge gap opens for U > Uc and they become 
insulating. 

The above results of D, n(k) and A^(q) indicate that in 4'q 
and a first-order metal-to-insulator transition occurs at 
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FIG. 9: (Color online) Optimized values of variational parameters in 
correlation factor V, for several t'/t as function of U/t; (a) g [onsite 
(Gutzwiller) correlation parameter], (b) fj, [doublon-holon binding 
parameter in the direction of t], and (c) /i' [the same of t']. For 
t'/t = 0.4-0.8, the parameters are optimized in the coexisting state 
*Q , and for t'/t = 1.0, in the 120° -AF state The symbols 

are common to all panels. 



U = Uc, as we showed for in (I). Nevertheless, the quan- 
tities studied below will show that these transitions do not be- 
long to pure Mott transitions with no relevance to magnetism 
like in ^fg, but to metal-to- magnetic-insulator transitions. 

Let us consider the optimized variational parameters in the 
correlation factor V. Shown in Figs. |9ja)-(c) is the U/t de- 
pendence of the optimized values of g, jj. and fi' for the lowest 
energy states: *J^°fort'/i = 0.4-0.8, and ^^^o f^^^y^ ^ q 
The fact that all the parameters show apparent discontinuities 
aXU = Uc supports the first-order transition. In comparing 
these values with the corresponding ones for shown in 
Fig. 4 in (1), we notice that the behavior of the Gutzwiller pa- 
rameter g is opposite near the critical point. At [/ = Uc, g for 
(i'/^ ^ 0-8) becomes larger in the insulating side U > Uc 
than in the metallic side [Fig.|9|a)], in contrast to the case for 
^'g [Fig. 4(a) in (1)]. This behavior can be understood reason- 
ably, if the (tt, 7r)-AF order arises in the insulating regime; it 
is known 13311 that g becomes larger in a projected (tt, 7r)-AF 
state than in the corresponding paramagnetic state, because 
the one-body Hartree-Fock state <i>AF already includes an ef- 
fect to suppress the double occupation, in inducing staggered 
spin structure. For 'I'g^ {t'/t = 1.0), the increase of g at 
Uc is still larger than that of '^q, meaning that the triplicate 
staggered field in <i>i2o forms a firmer order for the isotropic 
case. 



Another noticeable difference is the behavior of the 
doublon-holon binding parameter ii. The discontinuity of fi 
at Uc is an order of magnitude smaller in ^fg than in \l/g for 
t'/t < 0.8. This behavior is considered reasonable, again as- 
suming the (tt, ttV-AF order in the insulating regime. As we 
studied before. ll22ll the doublon-holon binding effect is intrin- 
sic in the Neel background of $af- Accordingly, in the cor- 
relation factor Vq plays a minor role for the (tt, 7r)-AF state. 
This tendency becomes more thorough for ^g"; fJ, in 5'g°, 
inversely, drops to almost zero at Uc and remains very small 
for U > Uc- Similarly, fi' drops to almost zero at Uc for 5'g °, 
and also for \l/g . Thus, the doublon-holon binding factor is 
almost useless for ^'g in the insulating regime. However, 
in the insulating regime of ^'g doublons exist as shown in 
Fig. |6l and we have confirmed in the records of Monte Carlo 
sweeps that a doublon almost necessarily sits in a nearest- 
neighbor site of a holon. This indicates that the one-body HF 
state ^'g " akeady has a sufficient doublon-holon binding ef- 
fect for finite A120. At any rate, the binding (and unbinding) 
of a doublon to a holon must be the essence of Mott transi- 
tions. 

To directly confirm the existence of long-range magnetic 
orders for U > Uc, we next discuss the behavior of the gap 
parameters, Aaf and A120, and the order parameter mg. For 
^'q , the sublattice magnetization mg is given, as usual, by 



(21) 



j 

with 5*1 = 1/2 ^c^ ^Cj.| — I'^j-lj ■ Similarly, we define m. 



for *g " as. 



TO, 



120 



1 



(22) 



with Sj' 



1/2 (at^^a,, T - For ml^° > 0, vf-^^o 

has a 120°-AF order In Figs. [TOfa) and (b), we show Aaf 
and nis of 5'g for three values of t'/t (< 0.8). The behavior 
of these two quantities is similar; they are negligibly small 
for U < Uc, whereas they abruptly increase at U = Uc and 
preserve the large magnitude for U > Uc- They are almost 
independent of the value of t'/t- We will turn to this point 
in nil CI Shown in Figs.[T0fc) and (d) are A120 and m^^^ of 
^'g" for t'/t — 1.0. Their U/t dependence is basically the 
same as those of 'I'g , but the magnitude of Ai2o/i and to^^" 
is larger than that of Aaf/^ and nis- In this point, the 120- 
degree AF order is not less steadfast than the (tt, 7r)-AF order 
The spin structure factor S'(q) is also checked (not shown), 
which has a sharp peak at q = {2tt/3, 27r/3) in the insulating 
regime of "fg^, supporting the realization of the 120° spin 
structure. Thus, we have confirmed that a firm magnetic long- 
range order always arises in the insulating regime at least for 
t'/t < 1- 

Finally, we discuss the d-wave gap A^ and the d-wave SC 
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FIG. 10: (Color online) (a) Optimized gap parameter AAp/t and (b) order parameter rris of a (7r,7r)-AF order for the coexisting state 
(t'/t = 0.4-0.8). (c) Optimized gap pai'ameter Ai2o/t and (d) order parameter ml^° of a 120° -AF order for the 120° -AF state *q° 
(t'/t = 1.0). For the full polarization, mg and mg^" become 1. 



correlation function of the nearest-neighbor-site pairing: 

■i t,t'— x.y 

(At(R,)A,,(R, + r)), (23) 

where x and y denote the lattice vectors in the x and y di- 
rections, and A|^(Ri) is the creation operator of a nearest- 
neighbor singlet. 

At (R,) = (4^4^^ + cl^^cl)/V2. (24) 

Unless Ad increases, Pd{r) does not increase, but the opposite 
does not hold, in contrast to the relation between Aaf and rris- 
It is possible that finite A^ indicates a non-SC singlet gap. 1 28] 
In contrast, Pd{r) is an good indicator of (i^2_j^2-wave SC, 
and was studied in detail for in (I), which yielded a con- 
clusion that SC arises for t'/t <^ 0.7 within ^tg. Here, we 
consider the long-distance behavior of Pd{v) by PJ'™, which 
is the average of P(i(r) only for r = {x,L/2) and {L/2,y) 
with x,y = 0-L. 

As shown in Fig. fTTT a). A^; for '^'q is always substantially 
zero for U < Uc- Accordingly, Pd(r) does not develop mean- 
ingfully exceeding the value of ?7 = 0, even if U approaches 
Uc, as shown in Figs.fTTlb) andfTTlc). This is in contrast with 
the case of 'i'g. Thus, appreciable SC does not appear in the 
conducting regime. In the insulating regime, the d-wave sin- 
glet gap Arf is still strongly suppressed in [Fig. fTTTa)]. 
compared with in '^'q [Fig. 4(c) in (I)], where Ad/t ^ 1.2- 

1.3. It is found, like the case of ^'q, Pd(r) is very small and 
vanishes rapidly as L increases (not shown). Consequently, 
for U > Uc, the (it, 7r)-AF order is overwhelmingly dominant 
over the d-wave SC order; in the insulating side can be re- 
garded as an almost pure (tt, 7r)-AF insulating state. It means 



that 'J'g undergoes a simple first-order metal-to-(7r, 7r)-AF- 
insulator transition atU = Uc [;29J for t'/t < 0.8. 

In conclusion, there is no chance that robust c?-wave SC or 
a nonmagnetic insulator appears within . 



C. Antiferromagnetic state and phase diagram 

In this subsection, we consider the properties of the (tt, tt)- 
AF state realized in the insulating regime of '^q. 

In (I), we found that the properties of in the (nonmag- 
netic) insulating regime are almost independent of the frustra- 
tion strength t'/t [cf. Fig. |4] for example]. This tendency be- 
comes more strong in ^'g . As in Fig. [T2l' b). the renormalized 
frustration t'/t becomes nearly zero for U > Uc, regardless 
of the model parameter t'/t, namely, in the strong coupling 
regime, the effective band almost retrieves the nesting con- 
dition for the siinple square lattice (t' — 0), even for highly 
frastrated cases. [30] The other variational parameters in \I>g 
are also almost independent of t'/t, as seen in each panel of 
Figs. |9l [Tola), [TTTa) and[T2ja), where all the data points for 
U > Uc are represented very well by a unique curve, regard- 
less of t'/t. Thus, the optimized ^'g is not changed with the 
frustration strength, as long as U > Uc- 

In Fig.m the total energy for ^fg in the insulating regime 
(U/t = 12) is plotted as a function of t'/t. Here, E"" is al- 
most constant, and the difference of E/t between t'/t — 
and 1.2 is as small as 0.1%. This behavior is not trivial even 
if the wave function is not changed with t'/t, because the t'- 
term in the Hamiltonian changes. To understand this result, 
we check the behavior of energy components; let Et, Et' and 
Eij be the contributions from the hopping in the t-bond and 
t'-bond directions, and from the onsite interaction U , respec- 
tively. We list the numerical data for t'/t = 0.8 in TableUas 
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FIG. 11: (Color online) (a) Optimized values of d-wave gap pa- 
rameter in ^^q" for t'/t — 0.4-0.8 as a function of U/t. Averaged 
nearest-neighbor d-wave pairing correlation function in ^g" for (b) 
t'/t = 0.4 and (c) t'/t = 0.8. Note that we average P<i(r) only for 
large values of |r| (see text). For U/t — 0, we use analytic values. 
The error bars in (b) and (c) include the standard deviations both of 
VMC calculations and by averaging with respect to r. 
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FIG. 12: (Color online) Optimized values of the remaining varia- 
tional parameters in vfo" for t'/t = 0.4-0.8 as a function of U /t; (a) 
C,/t [chemical potential], and (b) t'/t [band renormalization factor]. 
The symbols are conmion in both panels. 



a typical example, because each contribution is again almost 
constant as a function of t' /i. As expected, Ef is substan- 
tially zero, indicating if we allow the band renormalization, 
the wave function is by far stabilized by retrieving the nest- 
ing condition for the simple square lattice at the cost of the 
energy reduction due to the diagonal hopping or frustration, 
even \ft'/t is considerably large. 

It is natural to guess that this renormalization readily occurs 
in , because the nesting condition is advantageous not only 



TABLE I: Energy components and total energy of vpg for three 
values of U/t in the regime of the (tt, 7r)-AF insulator {U > Uc)- 
Here, t'/t = 0.8 {Uc/t ~ 6.65). The small system-size dependence 
is a characteristic of an (tt, 7r)-AF state. [33] The digits in brackets 
indicate the errors in the last digits. 



U/t 
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Et/t 


Et>/t 


Eu/t 


E/t 


10 


10 


-0.7761(6) 


-0.0001(0) 


0.3659(6) 


-0.4103(1) 




12 


-0.7759(9) 


-0.0001(0) 


0.3657(9) 


-0.4103(1) 


12 


10 


-0.6618(7) 


-0.0002(0) 


0.3134(7) 


-0.3485(1) 
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FIG. 13: (Color online) Ground-state phase diagram in the t'-U plane 
constructed from the present VMC results of the coexisting wave 
function *q and the 120°-AF state *q°. At the boundaries of the 
metallic and insulating phases, first-order magnetic transitions take 
place. 



to the (tt, 7r)-AF state but to the rf-wave state, as discussed in 
(1). Anyway, in recalling the point (iv) itemized in [III we no- 
tice that the band renormalization effect, namely the recovery 
of nesting, is essential to stabilize the (tt, 7r)-AF state, as well 
as the d-wave singlet state. ll23ll 

Finally, we discuss the ground-state phase diagram, which 
is reconstructed within and ° and depicted in Fig. [13] 
As compared with the diagram by '^'q and shown in 
Fig. 14 in (1), the area of the (tt, 7r)-AF insulator extends to 
extremely large t'/t (> 0.9) and to somewhat small U/t. 
In addition, the area of the 120°-AF insulator appears near 
the isotropic point t'/t — 1. We consider these tendencies 
are broadly consistent with the results for the J -J' model 
{U /t ~ oo), ||1Q,] in which the domain of (tt, 7r)-AF contin- 
ues to t'/t > 0.8. In Fig.[T3] as U /t increases, the boundary 
value in t'/t between the (tt, 7r)-AF and 120°-AF insulators 
tends to increase. This is probably because '^'q is stabilized 
by the d-wave gap A^, which rapidly increases for large U/t, 
as seen in Fig. fTTT a). We consider that the above tendency 
of the boundary will be corrected by introducing an appropri- 
ate singlet gap also into ^'q As a result of the stabilization 
of magnetic phases, the domains of nonmagnetic insulating 
and of robust rf-wave SC phases disappear, which occupy cer- 
tain parts of the phase diagram made in (1) and also in recent 
studies of a variational cluster perturbation theory [6] and a 



10 



(a)rf, ^2 (b) ext.s+t/jj, (c)rf, y+(rf^ | 




FIG. 14: Magnitude of pairing potentials |Ak/Aniax| considered 
in BCS state: (a) d^2_y2, (b) ext.s + dxy, and (c) d^2„y2 + idxy. 
|Aniax| denotes the maximum of | Ak| for each pairing gap. 



12. Thus, the singlet gap has an almost pure d^y-wave of one- 
dimensional character. As shown in Fig. 21 E^^''- overcomes 
E^^" for t' ^ ^ l-65< for U/t = 12, meaning that ^-^^o 
is predominant for an unexpectedly large range oft'/t (> 1) 
within the states we have studied {L = 10 and 12). We expect 
a more favorable pairing gap will be found for t < t' < 
but we leave a search for it for the future. 

Detailed results for ^'q^'' was reported in another publica- 
tion. H 

IV. CONCLUSION 



cellular dynamical mean field theory. |0] 



D. Extention of pairing-gap form 



From the argument in llllCI we expect a state yielding a gain 
in Et' overcomes and " for large t' /t. In this subsec- 
tion, we consider a couple of different pairing gaps, which 
seem suitable for t' ^ t, in the projected BCS function. 

One has a specific gap parameter to the t' direction (A^'), 
independent of for the t direction, lf32l[34ll 

Ak = As{coskx + cosky) — A^;' cos(fcj. + ky), (25) 

which we call "ext.sH-dxy wave" i^"^'^' = V'^s+d')- This 
form of Ak has nodes near the k^ and ky axes for Ag ^ A^/ 
[see Fig. [T4T b)l. which resembles the nodes proposed by some 
experiments. |,35>,361 Ak approaches the d^y wave of a one- 
dimensional character for jA^'l ^ \^s\- The other is a 
d^2_y2+id^y wave (*^+''' = V<i>d+td), 

r ■ 2tv ■ 4iT 1 

Ak = Ad+,;d coskx + e^~cos{kx + ky) + e^~cosky , (26) 

as shown in Fig. [T4T c). This form was often used to study fa- 
vorable gap symmetries for cobaltate SC; iIstI [ssi [39ll using a 
VMC method 1 39] for the t-J model on an isotropic triangular 
lattice, it was shown that vl/g+*'' is degenerate with at half 
filling, and has lower energy for doped cases. This gap form 
breaks a time reversal symmetry. 

In Fig.H the total energies of 5'^+'^' {E''+'^') and 
^j^d+id-^ are plotted in addition to those mentioned earlier. For 
t' /t ^1.1, E'^^'^'^ is almost constant in the same reason as 
E''° and -E'^, whereas E'^^^'^ starts to decreases at t' /t ^ 1.1 
abruptly, because, there, the direction of band renormalization 
is reversed from i' /t ^ to F/t —> oo. Thus, the effective 
Fermi surface of \I/g+"* becomes quasi one dimensional for 

t' /t ^1.1. Similarly to E'^^'^'^, E^^*^ considerably decreases 
as t' /t increases. In the range of decreasing E, the energy 
reduction in both and ^q''^' is largely attributed to Ef . 

Especially in ^g^'' , the energy reduction is entirely owing 
to Ef, and the direction of band renormalization is t' /t 
oo; the optimized A^ is negligible (^ 0.54) compared to the 
optimized A'^ (~ 7.05), for U/t = 12, t' /t = 1.2, and L = 



A. Summary 

As a continuation of the preceding study (I), f?) we have 
studied the Hubbard model on anisotropic triangular lattices, 
eq. ([T]i, at half filling, using an optimization variational Monte 
Carlo method. We introduce two new trial wave functions: 
(i) A coexisting state of (tt, 7r)-AF and d-wave gaps, which 
allows for a band renormalization effect, '^'q, and (ii) a state 
with an AF order of 120° spin structure, ^'g °. Main results 
are summarized as follows: 

[1] First-order metal-to-insulator transitions occur in both 
^fg and 5'g" at smaller values of U/t than those of the d- 
wave state ^'g studied in the preceding paper (1). As a re- 
sult, the regime of robust d-wave SC found in (1) is covered 
with the domain of these states. The modified phase diagram 
within 5'g and ^fg " is shown in Fig. [13] 

[2] In the insulating regimes, ^fg and ^'g ° are consider- 
ably stable, compared with ^Pg, and magnetic long-range or- 
ders always exist for t' /t <^ 1.65. Thus, a domain of a non- 
magnetic insulator is not found for t' ^ t within the wave 
functions used this time. 

[3] In the insulating regime of 'fg , the realized state can be 
regarded as a pure (tt, 7r)-AF insulator, because the sublattice 
magnetization as well as the (7r,7r)-AF gap (Aaf) is robust, 
and the d-wave pairing correlation almost vanishes. In the 
optimized ^fg , the effective band is renormalized so greatly 
(t' — ^ 0), irrespective of t' /t, that the nesting condition for 
t' = is retrieved almost completely. Accordingly, the con- 
tribution of diagonal hopping energy vanishes even for large 
t'/t. 

[4] For t' ^ t, ^>q" becomes predominant (U > Uc), 
even though the effects of band renormalization and of co- 
existing singlet gaps are not considered. If these effects are 
introduced, the area of the 120°-AF order will somewhat ex- 
pands, although, at present, the area of the (tt, 7r)-AF order 
extends to as large as t' /t ^ 0.9. 

[5] For large values of t' (> <c2 ^ 1.65), the singlet pairing 
states with gaps oriented to the diagonal-bond direction over- 
come 5'g°. We speculate that another predominant singlet 
(and SC) state will be discovered for t < t' < tc2- 

We believe that the mechanisms of a Mott (conductive-to- 
nonmagnetic insulator) transition and of the (ij.2_y2-wave SC 
pursued in (I) fundamentally remain valid, if the magnetic or- 
ders are removed for some reasons. However, the ground-state 
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phase diagram for the model eq. ([T) is substantially modified 
by and ^-^^o 

B. Discussions 

In comparing the present results with experimental ones of 
K-ET salts, a favorable point is that a (tt, 7r)-AF insulator is 
realized for realistic values of t' /t, namely e.g. 0.74 in n- 
(ET)2CuN(CN)2Cl. An unfavorable point is that robust SC 
and a nonmagnetic insulator do not appear; the latter state 
is believed to be realized in K-(ET)2Cu2(CN)3.||9t] One con- 
ceivable cause of this discrepancy is the insufficiency of trial 
wave functions; it is possible that quantum fluctuation is not 
sufficient for U ^ and large t' /t, and that we have not 
exhausted crucial orders. Another possible cause is that the 
present model eq. ([T]i is not sufficient to describe k-ET salts. 
For instance, the dimerization of ET molecules is not strong 
enough to justify the use of a single-band model. 1,41 1 

In the theoretical point of view, the present result is com- 
parable to that for U /t oo, namely the J- J' Heisenberg 
model. According to it, the (tt, 7r)-AF long-range order van- 
ishes citt'/t - 0.8, iSl and an AF order with 120° spin 
structure prevails at t' /t = l. lfTlll although a disordered phase 
may intervene between the two magnetic phases. Some other 
theoretical studies H S S il for the equivalent Hubbard 
model have yielded results of nonmagnetic insulating states 
at t' /t ~ 1. However, these studies have not explicitly treated 
the 120°-AF order, which is shown very stable for t' /t = 1 in 
this study. 



Although robust SC does not appear within the present 
study, we found that the symmetry of a singlet gap changes 
at large t' /t (~ 1.2) from the simple dx2_y2 wave to, for in- 
stance, the dxy wave as mentioned in llllDK see Fig. |4]l. This 
aspect is in accordance with that of FLEX, li4Zl in which 
a predominant SC symmetry switches from a ci^2_y2-wave 
to a da^y-wave state at t' /t ^ 1. Owing to this competi- 
tion between dx2^y2 and d^y waves near the isotropic point 
(t' /t — 1), the SC gap symmetry realized in k-ET salts, espe- 
cially in K-(ET)2Cu2(CN)3, may not be definitive but sensitive 
to physical parameters such as pressure. In contrast, a recent 
study of the Hubbard model with an extra exchange term using 
a Gutzwiller approximation |43] concluded that a d+id-wawe 
SC is stable for U ^ W and t' ^ t. Thus, it is urgent to carry 
out VMC calculations, in which the form of the pairing gap 
can be optimized without biased assumptions. 
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